Skip to content

DNM: h200 single-node sweep through srt-slurm launcher: docker - #3942

Draft
cquil11 wants to merge 17 commits into
mainfrom
dnm/h200-docker-launcher
Draft

DNM: h200 single-node sweep through srt-slurm launcher: docker#3942
cquil11 wants to merge 17 commits into
mainfrom
dnm/h200-docker-launcher

Conversation

@cquil11

@cquil11 cquil11 commented Oct 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Do not merge. Exercises srt-slurm's launcher: docker (SemiAnalysisAI/srt-slurm#47) end to end through Run Sweep on h200-dgxc, single node.

How it runs

  • utils/srt-slurm points at SemiAnalysisAI/srt-slurm dnm/inferencex-docker-h200: vllm cache reuse #47 merged with the pinned 545233a0, plus one fix (the job's own -e values now win over docker_args).
  • New cluster setting srt-slurm.single-node-launcher: docker, set on h200-dgxc. A single-node point then holds one whole node with an exclusive salloc, docker pulls the image there, and runs srtctl apply (launcher: docker) as a step on that node. srtctl runs the job in the foreground, so the step's exit code is the job's.
  • Pyxis hands a container the whole submit environment; docker gets only what it is told. The launch environment, minus host-specific names (paths, Slurm, GitHub/runner, locale, ...), is forwarded with docker_args: -e NAME, plus -w /infmax-workspace.
  • Outputs are chowned back to the runner user afterwards (the containers run as root).
  • Test-only edits: the configs below are cut to one or two points, AgentX duration is 1200 s, the c32 variants of the two dsv41flash recipes run behind a router frontend, and the two dsv41flash images are swapped to nightlies still on Docker Hub (the pinned tags were pruned).

Runs

Config Engine Exercises Result
dsr1 fp8 8k1k TP8 c4 SGLang fixed-seq-len pass
qwen3.5 fp8 TP8/EP8 MTP c8 SGLang AgentX, MTP pass
qwen3.5 fp8 TP8/EP8 MTP c8, hicache DRAM SGLang KV offload to CPU DRAM pass
minimaxm3 fp8 TP8 MTP c8 vLLM AgentX, MTP pass
qwen3.8next fp8 TP4/EP4 MTP c8 SGLang 4 of 8 GPUs, EP pass
dsv41flash fp4 TP4 DSpark c16 SGLang 4 of 8 GPUs, DSpark pass
dsv41flash fp4 TP4 DSpark c32, sglang-router SGLang separate frontend container pass
dsv41flash fp4 TP4 DSpark c32, vllm-router vLLM separate frontend container, setup_script pass
dsv41flash fp4 TP4 DSpark c16 vLLM 4 of 8 GPUs, DSpark fail: AgentX client Can not write request body in warmup; server up, one request running and ~21 queued on the swapped-in nightly

Failures along the way that were fixed or are not launcher issues:

  • First AgentX attempt: benchmark container missing IS_MULTINODE / AIPERF_*; fixed by forwarding the launch environment.
  • dsv41flash vLLM and dsv41flash SGLang: pinned nightly tags no longer on Docker Hub (h200 pyxis uses pre-imported squash files, so it never noticed).
  • dsv41flash SGLang attempt 1 of 38349058569: HuggingFace 429 loading the AgentX trace dataset; passed on rerun.

@cquil11
DNM: run h200 single-node points with srt-slurm launcher: docker
5be7dc6 
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
@cquil11
DNM: perf-changelog entry for the h200 docker launcher test
5afb2fe 
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
@github-actions

github-actions Bot commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献！

  • **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
  • **PR 验证：**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目，并且恰好带有一个主标签时，才会运行扫描：full-sweep-fail-fast（强烈推荐；canary 加逐矩阵 fail-fast）、full-sweep-enabled（有 canary；矩阵任务在失败后继续运行）或 non-canary-full-sweep-enabled（无 canary，也无 fail-fast）。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

@cquil11 cquil11 added the non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) label Oct 13, 2026
@github-actions

github-actions Bot commented Oct 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=38349058569

View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=38349058569

cquil11 added 11 commits October 12, 2026 19:55
@cquil11
DNM: forward the container environment pyxis would have passed to docker
c7c087f 
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
@cquil11
DNM: 20-minute AgentX points on h200 through launcher: docker
3d65f8c 
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
@cquil11
DNM: AgentX only, one config per run (qwen3.5 SGLang first)
0b55981 
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
@cquil11
DNM: router-frontend variants and one-point configs for the h200 dock…
1d73f26 
…er tests

Signed-off-by: Cam Quilici <cjquilici@gmail.com>
@cquil11
DNM: AgentX docker test: minimaxm3-fp8-h200-vllm-agentic-mtp
77ff0db 
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
@cquil11
DNM: forward the launch environment to docker as pyxis would
f7664f2 
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
@cquil11
DNM: AgentX docker test: qwen3.5-fp8-h200-sglang-agentic-mtp
5686404 
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
@cquil11
DNM: AgentX docker test: minimaxm3-fp8-h200-vllm-agentic-mtp
996c8f5 
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
@cquil11
DNM: AgentX docker test: dsv41flash-fp4-h200-vllm-agentic-dspark
044c579 
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
@cquil11
Merge remote-tracking branch 'origin/main' into dnm/h200-docker-launcher
ae6fa8c 
# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
@cquil11
DNM: h200 dsv41flash vLLM on a nightly still on Docker Hub
b8cfd04 
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
functionstackx
functionstackx requested changes Oct 13, 2026
View reviewed changes

@functionstackx functionstackx left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the need for this

@cquil11

cquil11 commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator Author

what is the need for this

W proactive /goal to make inferencex and set slurm more W

@cquil11
DNM: AgentX docker test: dsv41flash-fp4-h200-sglang-agentic-dspark
1478bfc 
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
@cquil11
DNM: AgentX docker test: qwen3.8next-fp8-h200-sglang-agentic-mtp
6772aa4 
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
@cquil11
DNM: h200 dsv41flash SGLang on a nightly still on Docker Hub
380035d 
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
@cquil11
DNM: AgentX docker test: dsv41flash-fp4-h200-sglang-agentic-dspark
c6bd9a9 
Signed-off-by: Cam Quilici <cjquilici@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@functionstackx functionstackx functionstackx requested changes
@Ankur-singh Ankur-singh Awaiting requested review from Ankur-singh Ankur-singh will be requested when the pull request is marked ready for review Ankur-singh is a code owner
@kedarpotdar-nv kedarpotdar-nv Awaiting requested review from kedarpotdar-nv kedarpotdar-nv will be requested when the pull request is marked ready for review kedarpotdar-nv is a code owner
@wzhao18 wzhao18 Awaiting requested review from wzhao18 wzhao18 will be requested when the pull request is marked ready for review wzhao18 is a code owner
@nvpohanh nvpohanh Awaiting requested review from nvpohanh nvpohanh will be requested when the pull request is marked ready for review nvpohanh is a code owner
@RohitNagraj RohitNagraj Awaiting requested review from RohitNagraj RohitNagraj will be requested when the pull request is marked ready for review RohitNagraj is a code owner

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim)

Projects

InferenceX Board
Status: No status

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@cquil11 @functionstackx